refactor: Postgres (+ Redshift) batch exports now async #18631
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tomasfarias
force-pushed
the
refactor/async-postgres-and-redshift-batch-exports
branch
from
5f78c22 to
ee69926
Compare
|
Going to merge this as the Redshift code includes a bugfix that we need to migrate folks over from export apps. |
tomasfarias
deleted the
refactor/async-postgres-and-redshift-batch-exports
branch
pauldambra
pushed a commit
that referenced
this pull request
Nov 15, 2023
* refactor: Postgres batch exports now async * fix: Let's use from psycopg import sql instead * test: Update Redshift tests * fix: Typing issues * fix: Main insert batch loop
This was referenced Nov 17, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Making PG batch exports async means less blocking the main thread, more concurrent tasks, more performance! Let's do all destinations async and bring down the number of required workers!
Turns out is quite easy to make Postgres batch exports async. What's more, the new COPY API doesn't require a file, so we may try a different buffered approach in the future. I'm very excited for the future of batch exports!
Changes
asyncandawait.""instead ofNone, but I don't think that's very significant: at least semantically we are not losing or gaining anything as these fields are meant to be deprecated, and in terms of schema both are compatible.👉 Stay up-to-date with PostHog coding conventions for a smoother review.
How did you test this code?
Same tests as before, just now async too.
Also added redshift tests that target local postgres db instance. Since postgres is fairly compatible with Redshift, having automated tests is better than nothing (of course, tests can still be run against a real Redshift instance).